The Assessment of Knowledge, in Theory and in Practice

This paper is adapted from a book and many scholarly articles. It reviews the main ideas of a novel theory for the assessment of a student’s knowledge in a topic and gives details on a practical implementation in the form of a software system available on the Internet. The basic concept of the theory is the ‘knowledge state,’ which is the complete set of problems that an individual is capable of solving in a particular topic, such as Arithmetic or Elementary Algebra. The task of the assessor—which is always a computer—consists in uncovering the particular state of the student being assessed, among all the feasible states. Even though the number of knowledge states for a topic may exceed several hundred thousand, these large numbers are well within the capacity of current home or school computers. The result of an assessment consists in two short lists of problems which may be labelled: ‘What the student can do’ and ‘What the student is ready to learn.’ In the most important applications of the theory, these two lists specify the exact knowledge state of the individual being assessed. This work is presented against the contrasting background of common methods of assessing human competence through standardized tests providing numerical scores. The philosophy of these methods, and their scientific origin in nineteenth century physics, are briefly examined. The assessment of human competence, as it is still performed today by many specialists in the schools and in the workplace, is almost systematically based on the numerical evaluation of some ‘aptitude.’ Its philosophy owes much to nineteenth century physics, whose methods were regarded as exemplary. The success of classical physics was certainly grounded in its use of a number of fundamental numerical scales, such as mass, time, or length, to describe basic aspects of objects or phenomena. In time, ‘measurement’ came to represent the sine qua non for precision and the essence of the scientific method, and physics the model for other sciences to imitate. In other words, for an academic endeavor to be called a ‘science,’ it had to resemble physics in critical ways. In particular, its basic observations had to be quantified in terms of measurement scales in the exact sense of classical physics. ∗Send correspondence to: Jean-Claude Falmagne, Dept. of Cognitive Sciences, University of California, Irvine, CA 92697. Phone: (949) 824 4880; FAX: (949) 824 1670; e-mail: [email protected]. We wish to thank Chris Doble, Dina Falmagne, and Lin Nutile for their reactions to earlier drafts of this article. Prominent advocates of this view were Francis Galton, Karl Pearson and WilliamThomson Kelvin. Because that position is still influential today, with a detrimental effect on fields such as ‘psychological measurement,’ which is relevant to our subject, it is worth quoting some opinions in detail. In Pearson’s biography of Galton (Pearson [1924, Vol. II, p. 345]), we find the following definition: “Anthropometry, or the art of measuring the physical and mental faculties of human beings, enables a shorthand description of any individual by measuring a small sample of his dimensions and qualities. This will sufficiently define his bodily proportions, his massiveness, strength, agility, keenness of senses, energy, health, intellectual capacity and mental character, and will constitute concise and exact numerical1 values for verbose and disputable estimates2.” For scientists of that era, it was hard to imagine a non-numerical approach to precise study of an empirical phenomenon. Karl Pearson himself, for instance—commenting on a piece critical of Galton’s methods by the editor of the Spectator3—, wrote “There might be difficulty in ranking Gladstone and Disraeli for ‘candour,’ but few would question John Morley’s position relative to both of them in this quality. It would require an intellect their equal to rank truly in scholarship Henry Bradshaw, Robertson Smith and Lord Acton, but most judges would place all three above Sir John Seeley, as they would place Seeley above Oscar Browning. After all, there are such things as brackets, which only makes the statistical theory of ranking slightly less simple in the handling.” (Pearson [1924, Vol. II, p. 345].) In other words, measuring a psychical attribute such as ‘candor’ only requires fudging a little around the edges of the order relation of the real numbers. The point here is that real numbers are still used to represent ‘quantity of attribute.’ As for Kelvin, his position on the subject is well known, and often represented in the form: “If you cannot measure it, then it is not science.” The full quotation is: “When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you are scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.” (Kelvin [1889].) Our emphasis. This excerpt is from an address “Anthropometry at Schools” given in 1905 by Galton at the London Congress of the Royal Institute for Preventive Medicine. The text was published in the Journal for Preventive Medicine, Vol. XIV, p. 93-98, London, 1906. The Spectator, May 23, 1874. The editor was taking Galton to task for his method of ranking applied to psychical character. He used ‘candour’ and ‘power of repartee’ as examples. Making such a relation a ‘weak order’ or perhaps a ‘semiorder’ (in the current terminology of combinatorics). A binary relation on a finite or countable set S is a weak order if there is a real valued function f defined on S such that x y ⇔ f(x) ≤ f(y) for all objects x and y in the set S. The relation is a semiorder if the representation has the form: x y ⇔ f(x) +1 ≤ f(y). For these concepts, see e.g. Roberts [1979] or Trotter [1992].